Papers with language modeling perplexity
Neural Syntactic Generative Models with Exact Marginalization (N18-1)
Copied to clipboard
| Challenge: | Recent models have added structure to recurrent neural networks at the cost of giving up exact inference, or using soft structure instead of latent variables. |
| Approach: | They propose a syntactic generative model with exact marginalization that supports dependency parsing and language modeling. |
| Outcome: | The proposed models achieve state-of-the-art for supervised dependency parsing and language modeling. |
CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have found that information relevant to the next token prediction task accumulates in the hidden representations of just a few tokens. |
| Approach: | They propose a method that integrates attention preferences useful for a downstream task into the eviction process of hidden states. |
| Outcome: | The proposed method performs better on comprehension and retrieval tasks while preserving language modeling perplexity. |
Meta-Learning Fast Weight Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens. |
| Approach: | They propose a neural component that uses gradient updates as linear attention to improve model performance. |
| Outcome: | The proposed model can be applied at training time and learn to make good use of gradient updates. |